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SUMMARY  AND  CONCLUSIONS 


Problem 


The  purpose  of  this  project  was  to  develop  an  instrument  useful  in  selec¬ 
ting  NROTC  (Regular)  students  with  maximum  likelihood  of  ultimately 
choosing  a  naval  career.  The  new  test  would  supplement  the  interviews 
and  the  tests  of  aptitude  and  interest  now  used. 

Background 


Two  earlier  NROTC  selection  instruments.  Forms  1  and  2  of  the  NROTC 
(Regular)  Questionnaire,  had  been  constructed  and  administered  in  1956- 
1958  to  applicants  for  NROTC  scholarships  for  whom  criterion  information 
(career  decision)  recently  became  available.  Forms  1  and  2  were  used  as 
an  item  pool  for  development  of  the  new  instrument. 

Approach 

The  above  forms  were  analyzed  and  the  questions  showing  the  greatest 
effectiveness  in  predicting  career  choice  over  an  eight-year  period  were 
identified  and  assembled  into  a  new  instrument,  the  U.  S.  Navy  Background 
Questionnaire.  Several  experimental  scoring  keys  were  constructed  and 
cross-validated  on  a  group  of  subjects  not  included  in  the  original  item 
analysis . 

Findings,  Conclusions,  Recommendations 

Analysis  of  the  data  indicated  the  new  form  to  have  sufficient  predictive 
validity  to  warrant  its  use  in  the  NROTC  selection  program.  As  more 
information  on  the  career  choice  of  NROTC  students  tested  during  1957- 
1963  becomes  available,  refinement  of  the  U.  S.  Navy  Background  Ques¬ 
tionnaire  will  be  possible  and  is  planned. 
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THE  DEVELOPMENT  OF  THE  U.  S.  NAVY  BACKGROUND  QUESTIONNAIRE 
FOR  NROTC  (REGULAR)  SELECTION 


A.  BACKGROUND  AND  PURPOSE 


Each  year  the  Navy  awards  some  1,400  NROTC  scholarships  to  qualified 
high  school  seniors,  entitling  each  recipient  to  a  four-year  college 
education,  which  includes  training  leading  toward  a  commission  as  a  naval 
officer.  Graduates  are  commissioned  in  the  Regular  Navy  or  Marine  Corps  and 
are  obligated  to  serve  on  active  duty  for  a  specified  length  of  time--four 
years  at  present.  The  scholarships  are  highly  desired,  as  evidenced  by  the 
approximately  20,000  applications  that  are  received  yearly,  and  thus  are 
extremely  competitive.  For  this  reason  and  because  of  the  substantial 
expense  to  the  Navy  in  sponsoring  this  program,  it  is  important  that  the 
selection  process  include  procedures  to  assess  the  applicants'  motivation 
toward  a  career  in  the  Navy.  To  this  end,  a  new  test,  the  U.  S.  Navy 
Background  Questionnaire,  was  constructed  for  use  in  the  1967  NROTC  applicant 
program  to  supplement  current  aptitude,  interest,  and  interview  information. 

Several  instruments  consisting  of  interest,  personality,  and  biographical 
items  had  been  developed  some  years  ago  in  an  attempt  to  measure  the  extent 
to  which  applicants  were  motivated  toward  a  naval  career.  Forms  1  and  2  of 
the  NROTC  Questionnaire,  the  instruments  designed  specifically  for  this 
purpose,  were  administered  as  part  of  the  NROTC  applicant  selection  programs 
conducted  from  1956  to  1963.  Since  obligatory  time  served  on  active  duty  has 
been  fulfilled  for  many  of  the  officers  administered  Forms  1  and  2,  it  was 
now  possible  to  assess  their  predictive  validity.  The  specific  purpose  of  this 
report  is  to  present  validation  procedures  and  results  for  the  Questionnaires, 
and  their  synthesis  into  the  U.  S.  Navy  Background  Questionnaire. 


B.  PROCEDURE 

1 .  Predictors 

A  brief  description  of  the  development  and  content  of  the  NROTC 
Questionnaire  (Forms  1  and  2)  is  provided  below. 

a.  NROTC  Questionnaire  (Form  1) .  Form  1  was  used  only  in  the  1956 
selection  program.  It  consisted  of  70  biographical  information  and  person¬ 
ality  items  followed  by  30  items  of  factual  knowledge  about  the  Navy,  ships, 
and  nautical  affairs  (Rimland,  1957) .  The  latter  30  items  comprise  the  Navy 
Knowledge  Test  (NKT)  (Rimland,  1959).  An  eight  year  follow-up  investigation 
into  the  validity  of  NKT  as  a  predictor  of  service  tenure  resulted  in  the  NKT 
not  being  recommended  for  use  in  NROTC  selection  (Githens,  Neumann  §  Abrahams, 
1966).  Since  other  studies  had  demonstrated  validity  for  direct  questions 
of  career  intention,  a  career  intention  question,  which  was  Item  13  in  the  70- 
item  personality  subtest  [to  be  referred  to  hereafter  as  the  Career  Question 
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(CQ)],  was  analyzed  separately  (Githens  §  Rimland,  1966).  Since  the  analysis 
confirmed  the  previous  studies,  it  was  suggested  that  future  NROTC  applicants 
be  asked  to  respond  to  a  direct  question  on  career  intention. 

b.  NROTC  Questionnaire  (Form  2) .  Form  2,  which  was  used  from  1957  to 
1963,  consisted  of  100  personality  and  biographical  type  items,  many  of 
which  were  taken  from  Form  1.  Eighteen  of  the  items  were  of  the  5-alternative 
multiple-choice  type,  while  the  remaining  82  items  were  in  diad  form.  A  more 
detailed  account  of  the  development  and  preliminary  validation  of  Form  2 
appears  in  an  earlier  report  (Rimland,  1957) .  Encouraging  results  were 
previously  obtained  with  a  career  motivation  scale  based  on  Form  2  items 
(Rimland  §  Steinemann,  1958) .  Several  groups  whose  level  of  career  moti¬ 
vation  was  inferable,  such  as  Academy  transfers  from  the  NROTC  program, 
voluntary  drops  from  the  NROTC  program,  and  NROTC  selected  applicants  were 
compared  on  mean  Form  2  scores.  Although  group  differences  were  not  large, 
they  were  in  the  expected  direction  and  recommendations  were  made  for  a 
longitudinal  investigation  of  the  validity  of  the  instrument,  to  supplement 
the  six  short-term  studies  which  had  been  made  during  the  test  development 
stage. 

2 .  Samples 

Included  in  the  present  validation  of  Forms  1  and  2  were  applicants  who 
(1)  accepted  a  Regular  NROTC  scholarship,  (2)  completed  either  Form  1  or 
Form  2  as  part  of  their  NROTC  selection  processing  in  1956,  1957,  or  1958, 

(3)  satisfactorily  completed  their  college  work  and  received  a  commission  in 
the  Regular  Navy,  and,  (4)  for  whom  retention  status  was  available.  Table  1 
presents  a  breakdown  of  sample  sizes  by  form  and  year  commissioned. 

Since  criterion  information  was  available  for  all  officers  in  the  Form  1 
sample,  a  total  of  767  questionnaires  were  available  for  analysis.  In  the 
Form  2  sample,  only  those  officers  who  were  commissioned  in  1961  had  reached 
a  point  of  career  decision.  Therefore,  the  number  of  Form  2  questionnaires 
that  could  be  analyzed  was  limited  to  397.  The  remaining  698  Form  2 
questionnaires  will  be  analyzed  at  a  later  date  as  the  career  decision  of  the 
officers  commissioned  in  1962  and  1963  becomes  known. 

3.  Criterion 


Potential  high  tenure  as  a  naval  officer  was  the  criterion  against 
which  the  two  questionnaires  were  validated.  At  least  six  months  of  active 
duty  beyond  the  original  obligated  service  time  was  designated  as  the 
requirement  for  inclusion  in  the  high  tenure  category.  Evaluations  based  on 
a  six-month  criterion  have  been  shown  to  hold  up  when  later  examined  using 
an  eighteen-month  criterion  (Githens  §  Rimland,  1966) .  For  those  entering 
the  NROTC  program  in  1956,  the  obligated  active  duty  time  from  date  of 
commissioning  was  three  years.  Beginning  with  the  year  1957  and  continuing 
until  the  present,  the  active  duty  obligation  time  was  increased  to  four 
years.  With  this  change  taken  into  account,  high  and  low  tenure  officers 
were  identified  in  both  samples. 
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TABLE  1 


Form  1  and  Form  2  Sample  Sizes  by  Year  Commissioned 


Questionnaire 

Year 

Commissioned 

N 

Form  1  (1956) a 

1959 

136 

1960 

469 

1961 

162 

Form  2  (1957-58) 

1961 

397 

1962 

536b 

1963 

162b 

Notes  -- 

cl 

Years  in  parentheses  refer  to  year  of  application. 
^Criterion  data  not  available  at  time  of  analyses. 


4 .  Analysis 

Since  the  procedures  followed  in  analyzing  Forms  1  and  2  were  essentially 
the  same,  the  ensuing  description  of  the  analyses  applies  to  both  question¬ 
naires.  After  the  sample  was  categorized  on  the  criterion,  a  portion  of  each 
category  was  randomly  selected  for  key  construction  purposes.  The  remainder 
was  set  aside  for  cross-validation  of  keys.  A  breakdown  for  each  question¬ 
naire  by  tenure  category  and  subsample  designation  is  presented  in  Table  2. 

Using  the  key  construction  sample,  differences  in  percentage  of 
endorsement  between  high  and  low  tenure  groups  were  determined  for  each  item 
response.  Several  keys  were  then  constructed  with  minimum  difference  levels 
for  item  response  inclusion  varying  from  6  to  13  per  cent.  Validities  for 
each  of  the  experimental  keys  were  obtained  for  the  appropriate  hold-out 
sample. 


C.  RESULTS  AND  DISCUSSION 


Table  3  presents  a  summary  of  results  in  the  cross-validation  sample 
for  the  three  experimental  Form  1  keys.  The  three  keys  differed  on  the 
minimum  percentage  difference  required  for  item  inclusion.  Minimum 
differences  of  6,  8,  and  10  per  cent  were  required  for  each  respective  key. 
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TABLE  2 


Number  of  Officers  in  Form  1  and  Form  2  Subsamples 
by  Tenure  Category 


Tenure 

Criterion 

Form  1 

Form  2 

Key 

Construction 

Cross- 

Validation 

Key 

Construction 

Cross- 

Validation 

High 

200 

109 

140 

87 

Low 

200 

258 

105 

65 

Totals 

400 

367 

245 

152 

For  all  three  keys,  biserial  validity  coefficients  were  low  but  were 
significantly  different  from  zero  on  cross-validation  (p  <  .05,  z  >  1.96). 

The  8-item  response  key  based  on  a  minimum  difference  of  10  per  cent  and 
the  14-item  response  key  based  on  a  minimum  difference  of  8  per  cent  resulted 
in  virtually  identical  validities.  Both  keys  were  slightly  superior  to  the 
6  per  cent  key  in  their  ability  to  discriminate  between  tenure  groups. 

While  the  validity  found  was  lower  than  had  been  hoped,  the  very  favorable 
selection  ratio  in  the  NROTC  program  tends  to  enhance  a  test's  usefulness 
in  selection. 

Predictive  validities  for  three  Form  2  experimental  keys  requiring 
minimum  percent  differences  of  13,  10,  and  8,  respectively,  are  presented  in 
Table  4.  A  percent  difference  criterion  of  13  provided  12  item  responses 
which  met  the  requirement.  Although  this  key  worked  as  well  as  or  better 
than  the  other  two  keys,  the  obtained  validity,  an  r^  of  .11,  is  not 
significant  at  the  .05  level.  As  the  criterion  matures  and  it  becomes 
possible  to  increase  the  sample  size  for  item  analysis  of  Form  2,  more  stable 
estimates  of  item  differences  can  be  expected. 

On  the  basis  of  the  foregoing  analyses,  the  1967  U.  S.  Navy  Background 
Questionnaire  was  constructed  for  aiding  in  the  selection  of  NROTC  scholarship 
recipients.  Items  from  both  forms  were  included- -eight  from  Form  1,  Key  B, 
and  31  from  Form  2,  Key  C.  Since  five  of  the  valid  items  appeared  on  both 
forms,  a  total  of  34  items  comprised  the  1967  Background  Questionnaire. 

Various  alternatives  were  available  in  selecting  the  most  efficient 
scoring  key  for  operational  use.  It  was  first  necessary  to  decide  whether 
the  operational  key  should  be  scored  on  Form  1  items  only.  Form  2  items  only, 
or  a  combination  of  items  from  both  forms.  Since  none  of  the  Form  2  key 
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validities  were  statistically  significant  on  the  cross-validation  sample,  they 
were  not  used  for  operational  scoring  at  this  time. 

It  was  necessary  to  choose  one  of  the  three  keys  based  on  Form  1.  Keys  A 
and  B  of  Form  1  were  equally  effective  in  classifying  the  sample  on  the  tenure 
criterion,  as  can  be  seen  in  Table  3.  Since  longer  keys  tend  to  have  higher 
reliability,  the  eight  per  cent  key  was  selected.  The  U.  S.  Navy  Background 
Questionnaire  was  first  used  operationally  in  January  1967,  as  part  of  the 
selection  battery  administered  to  NROTC  (Regular)  applicants. 

The  non-keyed  items,  all  of  which  have  demonstrated  potentially  useful 
validity,  were  retained  on  the  form  so  further  validation  information  could 
be  gathered  on  them. 

In  order  to  estimate  improvement  in  retention  provided  by  the  key,  the 
score  distributions  of  the  cross-validation  sample  were  used.  However,  since 
the  number  of  cases  and  the  retention  base  rate  in  the  cross-validation 
sample  differed  from  the  best  estimate  of  these  values,  which  is  the  total 
sample,  it  was  necessary  to  make  appropriate  adjustments.  Table  5  in  the 
Appendix  presents  distributions  for  both  actual  and  adjusted  frequencies  of 
the  cross-validation  sample. 

Use  of  the  adjusted  frequencies  yielded  a  total  of  110  officers  (14.3 
per  cent)  who  obtained  a  score  of  three  or  better  on  the  operational  key. 

Since  Form  1  influenced  selection  only  minimally  in  1956,  it  may  be  assumed 
that  scores  for  an  applicant  population  would  be  distributed  like  those  of 
the  present  sample.  If  this  assumption  is  made  and  only  the  top  14  per  cent 
of  the  applicants  (identified  by  the  operational  key)  were  selected,  then 
53.6  per  cent  would  be  expected  to  become  high  tenure  officers  (see  Figure  1) . 
This  would  represent  an  increase  of  13.3  percentage  points  over  the  base  rate 
of  40.3  per  cent  in  the  selected  population.  Since  such  a  stringent  cut-off 
would  probably  not  allow  for  a  sufficient  number  of  selectees  in  terms  of  the 
existing  selection  procedures,  other  cut-offs  must  be  considered.  If  the  cut¬ 
off  score  were  lowered  one  point,  to  include  all  those  with  a  score  of  two  or 
above,  approximately  24  per  cent  of  the  applicant  population  would  be 
available.  This  cut-off  yields  87  high  tenure  officers  from  a  possible  182 
(47.8  per  cent)  or  an  increase  of  7.5  percentage  points  over  the  base  rate. 

If  it  should  become  necessary  to  further  lower  the  cut-off  score  to  one  or 
better,  approximately  38  per  cent  of  the  population  would  be  selected  with 
the  resulting  increase  of  only  three  percentage  points  over  the  base  rate. 

The  effects  of  using  these  three  alternate  cut-offs  are  illustrated  in 
Figure  1. 


D.  CONCLUSIONS  AND  RECOMMENDATIONS 


1.  A  key  based  on  NROTC  (Regular)  Questionnaire  (Form  1)  was  found  to 
discriminate  significantly,  though  not  markedly,  between  high  and  low  tenure 
officers  in  cross-validation  samples. 
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Figure  1.  Percentage  of  High  Tenure  Officers  Expected  for  Various 

Cut-Offs  on  U.  S.  Navy  Background  Questionnaire  (Based  on 
Form  1  Score  Distributions) 


2.  A  similar  analysis  of  NROTC  (Regular)  Questionnaire  (Form  2)  did  not 
produce  a  significantly  valid  key  at  this  time.  Since  the  criterion  data 
were  available  for  only  a  portion  of  the  Form  2  sample,  it  is  recommended 
that  further  analyses  be  undertaken  when  a  larger  criterion  group  becomes 
available. 

3.  The  1967  U.  S.  Navy  Background  Questionnaire  was  developed  to  aid  in  the 
selection  of  NROTC  (Regular)  scholarship  recipients.  This  questionnaire 
was  composed  of  selected  items  from  both  Forms  1  and  2,  although  the 
operational  key  consisted  of  Form  1  items  only.  As  the  tenure  criterion 
matures  further,  it  is  recommended  that  the  present  key  be  enlarged  and 
updated . 

4.  Forms  1  and  2  may  prove  useful  in  predicting  criteria  other  than  career 
status.  Information  is  available  on  those  1956-1958  NROTC  (Regular) 
selectees  who  were  dropped  from  college  for  academic  reasons,  as  well  as 
for  those  who  do  not  receive  their  commissions  for  a  variety  of  other 
reasons.  Further  analyses  of  Forms  1  and  2  against  these  criterion  is 
indicated . 
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APPENDIX 


TABLE  5 


Score  Distributions  for  Actual  and  Adjusted  Frequencies 
of  Cross-Validation  Sample 


Score 

Actual 

cl 

Adjusted 

High 

Tenure 

Low 

Tenure 

High 

Tenure 

Low 

Tenure 

f 

cf 

f 

cf 

f 

cf 

f 

cf 

6 

1 

1 

1 

1 

3 

3 

2 

2 

5 

4 

5 

7 

8 

11 

14 

12 

14 

4 

6 

11 

9 

17 

17 

31 

16 

30 

3 

10 

21 

12 

29 

28 

59 

21 

51 

2 

10 

31 

25 

54 

28 

87 

44 

95 

1 

14 

45 

40 

94 

40 

127 

71 

166 

0 

25 

70 

37 

131 

71 

198 

66 

232 

-1 

14 

84 

38 

169 

40 

238 

68 

300 

-2 

8 

92 

31 

200 

23 

261 

55 

355 

-3 

11 

103 

22 

222 

31 

292 

39 

394 

-4 

3 

106 

22 

244 

8 

300 

39 

433 

-5 

2 

108 

9 

253 

6 

306 

16 

449 

-6 

1 

109 

4 

257 

3 

309 

7 

456 

-7 

0 

109 

1 

258 

0 

309 

2 

458 

Total  109 

258 

309 

458 

Note  -- 


a 


Adjusted  to  represent  base  rate  of  retention. 
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